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Background / Context: 

A variety of factors have converged in the last decade to focus attention on the need for more 
rigorous designs of professional development. Mounting evidence indicates that teachers differ 
substantially in their effectiveness (e.g., Aaronson, Barrow, & Sander, 2007; Rivkin, Hanushek, 
& Kain, 2005; Rockoff, 2004). And teacher development is increasingly viewed as one of the 
primary levers for improving teaching quality and ultimately student achievement (Correnti, 
2007; Desimone, 2009). This twin interest in teacher quality and teacher development has led 
major funding agencies to devote substantial resources to measuring and improving teacher 
quality and effectiveness. For instance, through many different programs and topics, the Institute 
of Education Sciences (IES) has funded over 60 projects that targeted the professional 
development of teachers and has recently established an entire program devoted to research on 
effective strategies for improving teacher quality through professional development (IES 
Education Research Grants, 2012). 

However, the scientific basis for improving development has been hampered by two major 
limitations. One is a lack of suitable instruments that provide valid, reliable and relevant 
measures of PD outcomes. Student measures are arguably too distal because many factors 
intervene between effective PD and what students leam (Yoon, Duncan, Lee, Scarloss, and 
Shapley, 2007). Conventional assessments of teacher content knowledge typically consist of 
straight subject matter tests and do not focus on the specialized types of content knowledge 
emphasized in PD and used in teaching (Ball, Thames & Phelps, 2010). Proxies such as teachers’ 
self-reports of their knowledge or learning do not assess what teachers actually know or leam 
(Garet, Porter, Desimone, Binnan, & Yoon, 2001; Desimone, Porter, Garet, Yoon, & Birman, 
2002 ). 

A more direct and proximal outcome of PD is the content knowledge that is typically the focus 
of PD and that teachers actually use in enacting effective teaching. Recent research has linked 
PD with changes in teachers' knowledge and teaching quality (Correnti, 2007; Garet et al., 2001). 
More recent literature has also established links among teacher knowledge and student learning 
in multiple subjects (Baumert, Kunter, Blum, 2010; Hill, Rowan, Ball, 2005; Kersting, Givvin, 
Thompson, et. al, 2012). Further, federal policy has acknowledged the importance of teachers' 
knowledge and its role in teachers' PD (Yoon et al., 2007), and IES has repeatedly identified 
teachers' knowledge as a valued outcome (e.g., IES Education Research Grants, 2012, p. 19). 

Until recently, there have been few instruments suitable for measures the types of teacher 
knowledge supported by the literature. However, new models of assessment are emerging that 
provide a direct measure of the content knowledge needed to address the content problems that 
arise in teaching. One prominent example are the Learning Mathematics for Teaching (LMT) 
measures developed by Ball, Hill, and colleagues (Hill, Schilling & Ball, 2004). There is strong 
evidence that the LMT assessments measure knowledge that is different from conventional tests 
of mathematics, specialized to teaching, sensitive to PD treatments, and associated with 
instructional quality and student outcomes (Hill et al., 2008; Rockoff, Jacob, Kane, Staiger, 

2011; Hill, Rowan & Ball, 2004; Hill, Schilling & Ball, 2004). 

A second limitation in the scientific basis or improving PD stems from the relative lack of 
relevant empirical estimates based new outcome assessments of teacher content knowledge. 
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Research focused on student learning outcomes has generated empirical estimates that can be 
used to appropriately estimate the power required for conducting group randomized trials 
(Hedges & Hedberg, 2007) as well as associated infonnation on average effect sizes (Bloom, 
2005; Jacob, Zhu, Bloom, 2010; Bloom, Zhu, Jacob, Raudenbush, Martineze & Lin, 2008). 
Research of this kind has just begun to estimate empirical parameters useful in designing 
appropriately powered group randomized trials that use teacher knowledge outcomes (Kelcey & 
Phelps, 2013a; Kelcey & Phelps, 2013b). However, there is limited information available on the 
average growth that researchers might expect to observe for teachers participating in PD. 

In this session, we present data from Teacher Knowledge Assessment System (TKAS), which is 
designed to administer the LMT measures. TKAS is being widely adopted in the evaluation of 
PD programs with over 500 separate program administrations and 16,000 teachers representing 
every major region in the country. TKAS provides a first of its kind database that can be used to 
assess the suitability of teacher knowledge assessments as tools for studying teacher 
development across a wide range of contexts, teachers and program designs. 

Purpose / Objective / Research Question / Focus of Study: 

Researchers interested in using teacher knowledge assessments as outcomes in the study of PD 
need relevant information about what kind of growth they might expect. Without a general sense 
of how teacher knowledge changes and how this growth varies by design relevant characteristics 
of teachers, teaching context, and features of PD itself, researchers are largely guessing at what 
might constitute expected and meaningful change in teacher knowledge. 

We will use the pre- to post-test gains in knowledge made by teachers over the course of their 
respective professional development programs to estimate empirical benchmarks for average 
teacher change across all PD program for each of the five teacher knowledge outcomes 
administered in TKAS. As a second step, we will select characteristics of teachers (degree 
training in mathematics, teaching experience), teaching environment (school SES, urbanicity and 
region), and basic design features of PD programs (the time of year that PD is conducted) that 
could act to moderate the average change in teacher knowledge pre- to post-test. 

Setting and Data Collection: 

The TKAS database includes 5 different elementary and middle school outcomes: (1) 

Elementary number and operations (ELNCOP); (2) Elementary patterns, functions and algebra 
(ELPFA); (3) grade 4-8 geometry (GEO); (4) Middle school number and operations (MSNCOP); 
(5) Middle school patterns, functions and algebra (MSPFA). Data comes from 41 states and the 
District of Columbia. While not nationally representative (or representative of states), these data 
comprise one of the largest samples of teacher PD programs to date. They also, given that the 
LMT assessments and TKAS are one of the only instruments available to use in pre- to post- test 
evaluations, are a viable sample for representing math PD that is involved in evaluation. 

TKAS is used by a variety of users (e.g., teacher educators, district personnel) for a variety of 
purposes. To ensure that our analytic sample only included teachers enrolled in professional 
development programs, we limited our sample by first dropping preservice teachers and 
preservice programs from the sample and by excluding PD programs that had not yet 
administered a post-test to participating teachers. 
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Our final sample for each assessment is shown in Table 1. As indicated in Table 1, a large 
number of teachers did not complete post assessments. For the preliminary analyses reported on 
below, we have included only teachers who had pre- and post- test data available by employing 
list wise deletion. We will conduct final analyses using appropriate multiple imputation 
accounting for the cross-classified and nested structure of our data. 

Research Design: 

Our research goals are to first generate estimates of average change in teacher knowledge across 
all programs and next to examine the effects of design-relevant moderator variables on this 
average change. Our data structure has teachers cross-classified within schools and programs 
(i.e., in some cases, teachers from more than one school appears in a given program and in other 
cases, teachers from a single school appear in more than one program). To appropriately account 
for this cross-nested structure, we ran cross-classified models as specified above. In a second 
stage of the analysis we run a series of separate random intercept models with each moderator 
variable entered one at a time, the goal of which was to estimate the effects of the moderator 
variables in the average pre- to post- test estimates. 


Statistical, Measurement, or Econometric Model: 

To estimate the average pre- to post-test change for teachers attending math PDs, we use a cross- 
classified model which takes into account the non-hierarchical structure that teachers in one PD 
could come from different schools and teacher in one school could go to different PDs. Then, we 
add moderators into the unconditional model to examine the extent to which this change from 
pre- to post- test score are affected by teacher characteristics, PD and school features. This 
procedure is repeated across each of the five study outcomes. The unconditional model is 
specified by equation 1 and 2 below 


Postscore ijk =/?„+/?, Pr escore ijk + s ijk 

Po = 8oo +r ij+ r 2 k +u ijk 

Or 


( 1 ) 

( 2 ) 


A = koo + 8oi W + r ij + r 2k + u ijk (3) 

Where gOO is the average change from pre- to post- test score PD, r l ; is the random effect for 


programs and r 2k is the random effect for schools. Equation 3 specifies the second level of the 

model when moderators are included, but the main level still remains the same. W in Equation 3 
could be a teacher, PD or school variable. The model is repeated for each variable of interest 
with only one added at a time. 


Significance / Novelty of study: 

The empirical benchmarks of average change in teacher knowledge serves two purposes. First, 
these benchmarks provide a context for interpreting the magnitude of effect sizes drawn from 
future empirical studies. In this sense, our results provide a more relevant point of reference than 
two other commonly-employed benchmarks — Cohen’s guidelines for small, medium, and large 
effect sizes, and empirical benchmarks that have been derived from studies using student 
outcomes. If studies include teacher knowledge as an outcome, the magnitude of their effects 
should be interpreted relative to other studies of teacher knowledge that use similar outcomes 
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and are conducted in similar contexts with similar teachers. Second, the results will provide 
guidance in the design of future studies of the effectiveness of PD interventions. As Desimone 
(2009) and others have argued, teacher knowledge is an important outcome in PD interventions, 
and it may provide a more proximal measure of a program’s effectiveness than student 
achievement. 

Findings / Results: 

The pre- to post- test changes for each assessment are summarized in Table 2 along with the 
variance components attributable to school, PD program and to teachers. The average change 
varies from 0.17 for EL PFA to 0.30 for 4-8 GEO. The variance between PD programs is larger 
than schools, however the majority of variance remains at the teacher level. The variance at each 
level is similar across the five outcomes. Figure 1 presents a helpful visual representation of the 
distribution of how change in outcome measures varies across programs and illustrates the 
relatively small proportion of programs where teachers are showing substantial gains of 0.50 SD 
or more. 

Tables 3-7 present the results for the moderator analysis for each outcome. One of the main 
outcomes of the work are the empirical estimates. Clearly, one limitation of these preliminary 
analysis is a limited sample size for a number of moderators, an issue that will be addressed in 
part when post scores are imputed for the roughly 50% of teachers missing at post test. 

However, across the tests there is sufficient sample size to note that a number of moderators are 
related to quite substantial differences in change. For example, having a math degree is 
constantly related to change that is 0.02 SD or more than teachers without a math degree. On the 
other hand, years of experience has different effects across different assessments, suggesting that 
this teacher characteristic is dependent on the outcome measure. 

Conclusions: 

These findings provide potential guidance for the design of studies using teacher knowledge as 
an outcome measure. In general, the estimates drawn from studies using teacher outcomes are 
smaller than those found in studies using student achievement as an outcome; they are also 
“small” based on the benchmarks put forward by Cohen. These findings suggest that studies 
using teacher knowledge as an outcome should be considered differently than those using 
measure of student achievement where larger effects might be expected. Most notable about our 
findings is the variation across outcomes, suggesting that researchers should consider the specific 
math outcome that is most relevant for their intervention. It is also noteworthy that design- 
relevant moderators have different effects on pre- to post-test change both within and across 
tests. While we anticipate that the coefficients will change after we complete analysis with 
appropriate imputation, we expect that the stronger trends noted above will be similar to what is 
shown in the preliminary analysis. These results provide context relevant guidelines (e.g., test 
outcome, characteristics of teachers, school or program) that researchers may use individually or 
in combination to design studies and/or assess the practical impact of teacher PD interventions. 
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Appendix B. Tables and Figures 


Table 1 

TKAS Sample for Each Outcome Measure 



Teacher Pre-Test 

Teacher Post-Test 

Program (Pre) 

School (Pre) 

EL NCOP 

4,313 

2,129 

112 

994 

EL PFA 

2,574 

1,253 

55 

612 

MS NCOP 

1,363 

831 

49 

503 

MS PFA 

1,785 

2,901 

83 

1,093 

4-8 GEO 

973 

652 

44 

349 
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Table 2 

Unconditional Estimates ofPre- to Post Test Change for Teacher Knowledge Outcomes 



Average 

Change 

Rj 2 

R k 2 

R 2 

J 

K 

N 

EL NCOP 

0.24 

0.04 

0.01 

0.44 

71 

505 

1,432 

EL PFA 

0.17 

0.08 

0.02 

0.55 

36 

299 

818 

MS NCOP 

0.24 

0.04 

0.03 

0.51 

33 

298 

549 

MS PFA 

0.20 

0.05 

0.01 

0.42 

58 

644 

1,173 

4-8 GEO 

\ t i r\ 2 1 r» 

0.30 

2 r* i i i 

0.08 

0.00 
n i v 

0.35 

ill 

31 

j • 1 r\ 

201 

2 r* 

390 

i i i 


2 2 7 ; 7 

Note. Rj and RC refer to the variance between PDs and schools, respectively. R, refers to the 
residual variance not due to additive effects of PDs and Schools. 
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Table 3 

Moderator Estimates ofPre- to Post Test Change for Elementary Number Concepts and 
Operations 



Average 

Change 

R j 2 

Rk 2 

Rf 

J 

K 

N 

Unconditional Model 

0.24 

0.04 

0.01 

0.44 

71 

505 

1,432 

Math degree 

Yes 

0.40 

0.04 

0.02 

0.44 

37 

84 

106 

No 

0.23 

0.04 

0.02 

0.44 

71 

476 

1,326 

Years of Experience 

0-3 

0.23 

0.04 

0.01 

0.44 

58 

154 

235 

4-15 

0.25 

0.04 

0.01 

0.44 

69 

371 

778 

>15 

0.23 

0.04 

0.01 

0.44 

62 

221 

419 

Program type 

Summer Institute 

0.29 

0.04 

0.01 

0.45 

15 

119 

327 

School year program 

0.20 

0.04 

0.01 

0.45 

25 

167 

580 

Both 

0.24 

0.04 

0.01 

0.45 

31 

234 

525 

% of free/reduced lunch 

0-0.25 

0.21 

0.04 

0.01 

0.45 

32 

77 

190 

0.26-0.50 

0.26 

0.04 

0.01 

0.45 

52 

165 

513 

0.51-0.75 

0.25 

0.04 

0.01 

0.45 

51 

153 

473 

0.76-1 

0.21 

0.04 

0.01 

0.45 

40 

106 

237 

Region 

NE 

0.45 

0.04 

0.02 

0.44 

3 

21 

31 

S 

0.20 

0.04 

0.02 

0.44 

19 

171 

510 

MW 

0.26 

0.04 

0.02 

0.44 

24 

125 

455 

W 

0.23 

0.04 

0.02 

0.44 

30 

188 

436 

Urban 

City 

0.29 

0.04 

0.01 

0.44 

36 

129 

330 

Suburb 

0.23 

0.04 

0.01 

0.44 

42 

149 

406 

Town 

0.19 

0.04 

0.01 

0.44 

33 

69 

170 

Rural 

0.23 

0.04 

0.01 

0.44 

55 

158 

526 


2 2 7 ; 2 

Note. Rj“ and Rk“ refer to the variance between PDs and schools, respectively. Rf refers to the 
residual variance not due to additive effects of PDs and Schools. 
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Table 4 

Moderator Estimates ofPre- to Post Test Change for Elementary Patterns Functions and 
Algebra 



Average 

Change 

R j 2 

Rk 2 

Rf 

J 

K 

N 

Unconditional Model 

0.17 

0.08 

0.02 

0.55 

36 

299 

818 

Math degree 

Yes 

0.44 

0.08 

0.02 

0.54 

20 

57 

75 

No 

0.14 

0.08 

0.02 

0.54 

36 

274 

743 

Years of Experience 

0-3 

0.20 

0.08 

0.02 

0.55 

24 

86 

124 

4-15 

0.19 

0.08 

0.02 

0.55 

34 

220 

478 

>15 

0.12 

0.08 

0.02 

0.55 

32 

135 

216 

Program type 

Summer Institute 

-0.04 

0.07 

0.02 

0.55 

3 

26 

123 

School year program 

0.11 

0.07 

0.02 

0.55 

8 

97 

300 

Both 

0.23 

0.07 

0.02 

0.55 

25 

176 

395 

% of free/reduced lunch 

0-0.25 

0.06 

0.08 

0.02 

0.55 

13 

23 

79 

0.26-0.50 

0.16 

0.08 

0.02 

0.55 

28 

87 

252 

0.51-0.75 

0.22 

0.08 

0.02 

0.55 

27 

118 

329 

0.76-1 

0.17 

0.08 

0.02 

0.55 

25 

69 

154 

Region 

NE 

- 

- 

- 

- 

0 

0 

0 

S 

0.09 

0.07 

0.02 

0.55 

21 

166 

495 

MW 

0.28 

0.07 

0.02 

0.55 

6 

64 

155 

W 

0.26 

0.07 

0.02 

0.55 

9 

69 

168 

Urban 

City 

0.18 

0.08 

0.02 

0.55 

22 

63 

158 

Suburb 

0.08 

0.08 

0.02 

0.55 

19 

62 

212 

Town 

0.23 

0.08 

0.02 

0.55 

21 

58 

129 

Rural 

0.18 

0.08 

0.02 

0.55 

26 

116 

319 


2 2 7 ; 7 

Note. Rf and Rk" refer to the variance between PDs and schools, respectively. Rf refers to the 
residual variance not due to additive effects of PDs and Schools. 
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Table 5 

Moderator Estimates ofPre- to Post Test Change for Middle School Number Concepts and 
Operations 



Average 

Change 

R j 2 

R k 2 

R , 2 

J 

K 

N 

Unconditional Model 

0.24 

0.04 

0.03 

0.51 

33 

298 

549 

Math degree 

Yes 

0.39 

0.04 

0.03 

0.49 

32 

147 

224 

No 

0.16 

0.04 

0.03 

0.49 

32 

212 

325 

Years of Experience 

0-3 

0.16 

0.04 

0.03 

0.51 

25 

99 

134 

4-15 

0.26 

0.04 

0.03 

0.51 

31 

205 

291 

>15 

0.29 

0.04 

0.03 

0.51 

25 

98 

124 

Program type 

Summer Institute 

0.17 

0.04 

0.03 

0.51 

2 

20 

27 

School year program 

0.27 

0.04 

0.03 

0.51 

9 

116 

263 

Both 

0.24 

0.04 

0.03 

0.51 

22 

164 

259 

% of free/reduced lunch 

0-0.25 

0.43 

0.03 

0.03 

0.50 

13 

28 

41 

0.26-0.50 

0.30 

0.03 

0.03 

0.50 

25 

101 

161 

0.51-0.75 

0.25 

0.03 

0.03 

0.50 

27 

109 

240 

0.76-1 

0.04 

0.03 

0.03 

0.50 

19 

58 

105 

Region 

NE 

- 

- 

- 

- 

0 

0 

0 

S 

0.32 

0.04 

0.02 

0.51 

17 

186 

374 

MW 

0.22 

0.04 

0.02 

0.51 

4 

29 

45 

W 

0.12 

0.04 

0.02 

0.51 

13 

83 

129 

Urban 

City 

0.38 

0.05 

0.01 

0.51 

19 

85 

183 

Suburb 

0.10 

0.05 

0.01 

0.51 

15 

48 

75 

Town 

0.24 

0.05 

0.01 

0.51 

15 

43 

83 

Rural 

0.23 

0.05 

0.01 

0.51 

28 

122 

208 


2 2 7 ; 7 

Note. Rj" and Rk“ refer to the variance between PDs and schools, respectively. Rf refers to the 
residual variance not due to additive effects of PDs and Schools. 
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Table 6 

Moderator Estimates ofPre- to Post Test Change for Middle School Patterns Functions and 
Algebra 



Average 

Change 

R j 2 

Rk 2 

Ri 2 

J 

K 

N 

Unconditional Model 

0.20 

0.05 

0.01 

0.42 

58 

644 

1173 

Math degree 

Yes 

0.33 

0.05 

0.00 

0.42 

54 

330 

489 

No 

0.12 

0.05 

0.00 

0.42 

57 

434 

684 

Years of Experience 

0-3 

0.09 

0.05 

0.01 

0.42 

52 

206 

290 

4-15 

0.23 

0.05 

0.01 

0.42 

55 

444 

648 

>15 

0.23 

0.05 

0.01 

0.42 

49 

194 

235 

Program type 

Summer Institute 

0.36 

0.05 

0.01 

0.42 

5 

54 

72 

School year program 

0.17 

0.05 

0.01 

0.42 

15 

280 

605 

Both 

0.19 

0.05 

0.01 

0.42 

38 

321 

496 

% of free/reduced lunch 

0-0.25 

0.22 

0.05 

0.01 

0.42 

23 

66 

95 

0.26-0.50 

0.23 

0.05 

0.01 

0.42 

46 

170 

269 

0.51-0.75 

0.19 

0.05 

0.01 

0.42 

48 

244 

494 

0.76-1 

0.17 

0.05 

0.01 

0.42 

36 

152 

302 

Region 

NE 

0.42 

0.05 

0.01 

0.42 

1 

10 

10 

S 

0.21 

0.05 

0.01 

0.42 

40 

489 

925 

MW 

0.10 

0.05 

0.01 

0.42 

5 

34 

54 

W 

0.20 

0.05 

0.01 

0.42 

15 

111 

183 

Urban 

City 

0.20 

0.05 

0.01 

0.42 

43 

218 

488 

Suburb 

0.15 

0.05 

0.01 

0.42 

22 

90 

133 

Town 

0.21 

0.05 

0.01 

0.42 

36 

105 

184 

Rural 

0.21 

0.05 

0.01 

0.42 

49 

229 

366 
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Note. Rj" and Rk" refer to the variance between PDs and schools, respectively. Rf refers to the 
residual variance not due to additive effects of PDs and Schools. 


Table 7 

Moderator Estimates ofPre- to Post Test Change for 4-8 Geometry 


Averag 



e 

Change 

Rj 2 

Rk 

Ri 

J 

K 

N 

Unconditional Model 

0.30 

0.08 

0.00 

0.35 

31 

201 

390 

Math degree 

Yes 

0.40 

0.06 

0.00 

0.35 

24 

64 

93 

No 

0.26 

0.06 

0.00 

0.35 

30 

162 

297 

Years of Experience 

0-3 

0.40 

0.07 

0.00 

0.34 

23 

50 

65 

4-15 

0.33 

0.07 

0.00 

0.34 

28 

141 

225 

>15 

0.16 

0.07 

0.00 

0.34 

27 

77 

100 

Program type 

Summer Institute 

- 

- 

- 

- 

0 

0 

0 

School year program 

0.41 

0.07 

0.00 

0.35 

9 

78 

107 

Both 

0.25 

0.07 

0.00 

0.35 

22 

137 

283 

% of free/reduced lunch 

0-0.25 

0.43 

0.06 

0.00 

0.35 

12 

25 

28 

0.26-0.50 

0.32 

0.06 

0.00 

0.35 

27 

74 

137 

0.51-0.75 

0.23 

0.06 

0.00 

0.35 

22 

71 

175 

0.76-1 

0.34 

0.06 

0.00 

0.35 

12 

29 

47 

Region 

NE 

- 

- 

- 

- 

0 

0 

0 

S 

0.30 

0.07 

0.00 

0.35 

21 

145 

286 

MW 

0.23 

0.07 

0.00 

0.35 

2 

31 

50 

W 

0.33 

0.07 

0.00 

0.35 

8 

25 

54 

Urban 
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City 

0.42 

0.07 

0.00 

0.34 

19 

43 

89 

Suburb 

0.33 

0.07 

0.00 

0.34 

14 

30 

55 

Town 

0.12 

0.07 

0.00 

0.34 

14 

40 

69 

Rural 

0.29 

0.07 

0.00 

0.34 

24 

88 

177 


2 2 7 7 

Note. Rf and Rk“ refer to the variance between PDs and schools, respectively. Rf refers to the 
residual variance not due to additive effects of PDs and Schools. 


Figure 1 

Distribution ofPre- to Post Test Professional Development Averages for Teacher Knowledge 
Outcomes 



Note: The Y axis is reported as a probability density. 
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